Echo state model of non-Markovian reinforcement learning, An

Department Head: Dale H. Grit.2008 Spring.Includes bibliographical references (pages 137-142).There exists a growing need for intelligent, autonomous control strategies that operate in real-world domains. Theoretically the state-action space must exhibit the Markov property in order for reinforcement learning to be applicable. Empirical evidence, however, suggests that reinforcement learning also applies to domains where the state-action space is approximately Markovian, a requirement for the overwhelming majority of real-world domains. These domains, termed non-Markovian reinforcement learning domains, raise a unique set of practical challenges. The reconstruction dimension required to approximate a Markovian state-space is unknown a priori and can potentially be large. Further, spatial complexity of local function approximation of the reinforcement learning domain grows exponentially with the reconstruction dimension. Parameterized dynamic systems alleviate both embedding length and state-space dimensionality concerns by reconstructing an approximate Markovian state-space via a compact, recurrent representation. Yet this representation extracts a cost; modeling reinforcement learning domains via adaptive, parameterized dynamic systems is characterized by instability, slow-convergence, and high computational or spatial training complexity. The objectives of this research are to demonstrate a stable, convergent, accurate, and scalable model of non-Markovian reinforcement learning domains. These objectives are fulfilled via fixed point analysis of the dynamics underlying the reinforcement learning domain and the Echo State Network, a class of parameterized dynamic system. Understanding models of non-Markovian reinforcement learning domains requires understanding the interactions between learning domains and their models. Fixed point analysis of the Mountain Car Problem reinforcement learning domain, for both local and nonlocal function approximations, suggests a close relationship between the locality of the approximation and the number and severity of bifurcations of the fixed point structure. This research suggests the likely cause of this relationship: reinforcement learning domains exist within a dynamic feature space in which trajectories are analogous to states. The fixed point structure maps dynamic space onto state-space. This explanation suggests two testable hypotheses. Reinforcement learning is sensitive to state-space locality because states cluster as trajectories in time rather than space. Second, models using trajectory-based features should exhibit good modeling performance and few changes in fixed point structure. Analysis of performance of lookup table, feedforward neural network, and Echo State Network (ESN) on the Mountain Car Problem reinforcement learning domain confirm these hypotheses. The ESN is a large, sparse, randomly-generated, unadapted recurrent neural network, which adapts a linear projection of the target domain onto the hidden layer. ESN modeling results on reinforcement learning domains show it achieves performance comparable to lookup table and neural network architectures on the Mountain Car Problem with minimal changes to fixed point structure. Also, the ESN achieves lookup table caliber performance when modeling Acrobot, a four-dimensional control problem, but is less successful modeling the lower dimensional Modified Mountain Car Problem. These performance discrepancies are attributed to the ESN’s excellent ability to represent complex short term dynamics, and its inability to consolidate long temporal dependencies into a static memory. Without memory consolidation, reinforcement learning domains exhibiting attractors with multiple dynamic scales are unlikely to be well-modeled via ESN. To mediate this problem, a simple ESN memory consolidation method is presented and tested for stationary dynamic systems. These results indicate the potential to improve modeling performance in reinforcement learning domains via memory consolidation

The corporatization of community pharmacy:implications for service provision, the public health function, and pharmacy's claims to professional status in the United Kingdom

Background Pharmacy has experienced both incomplete professionalization and deprofessionalization. Since the late 1970s, a concerted attempt has been made to re-professionalize pharmacy in the United Kingdom (UK) through role extension—a key feature of which has been a drive for greater pharmacy involvement in public health. However, the continual corporatization of the UK community pharmacy sector may reduce the professional autonomy of pharmacists and may threaten to constrain attempts at reprofessionalization. Objectives The objectives of the research: to examine the public health activities of community pharmacists in the UK; to explore the attitudes of community pharmacists toward recent relevant UK policy and barriers to the development of their public health function; and, to investigate associations between activity, attitudes, and the type of community pharmacy worked in (eg, supermarket, chain, independent). Methods A self-completion postal questionnaire was sent to a random sample of practicing community pharmacists, stratified for country and sex, within Great Britain (n = 1998), with a follow-up to nonresponders 4 weeks later. Data were analyzed using SPSS (SPSS Inc., Chicago, IL, USA) (v12.0). A final response rate of 51% (n = 1023/1998) was achieved. Results The level of provision of emergency hormonal contraception on a patient group direction, supervised administration of medicines, and needle-exchange schemes was lower in supermarket pharmacies than in the other types of pharmacy. Respondents believed that supermarkets and the major multiple pharmacy chains held an advantageous position in terms of attracting financing for service development despite suggesting that the premises of such pharmacies may not be the most suitable for the provision of such services. Conclusions A mixed market in community pharmacy may be required to maintain a comprehensive range of pharmacy-based public health services and provide maximum benefit to all patients. Longitudinal monitoring is recommended to ensure that service provision is adequate across the pharmacy network

Community pharmacists' engagement with public health in Great Britain

Objective - The objective of the research was to examine to what extent community pharmacists in Great Britain believed that their job was concerned with local public health issues. Methods - The project (Pharmacy and Public Health)received ethical approval from the Research Ethics Committee of the School of Life and Health Sciences at Aston University. After piloting, in August 2006 a self-completion postal questionnaire was sent to practicing community pharmacists in Great Britain (n=1998), with a follow-up to non-responders 4 weeks later. A final response rate of 51% (n=1023/1998) was achieved. Results - Respondents were asked to indicate their answer to the question “to what extent is your present job concerned with local public health issues?” on a three-point scale – “highly”, “slightly” or “not at all” concerned with public health. They were also asked to indicate whether they were pharmacy owners, employee pharmacists or self-employed locum pharmacists. Less than half (43%,n=384/898) of respondents answering both questions believed that their job was highly concerned with public health. A relationship was observed between employment status and the level to which a respondent believed that their job was concerned with public health (chi-square test with P=0.001). Over half of pharmacy owners (51%, n=68/134) considered that their job was highly concerned with public health compared to44% (n=193/443) of employee pharmacists and38% (n=123/321) of locum pharmacists. Conclusion - This research suggests that community pharmacists in Great Britain are not ‘fully engaged’ with public health. Pharmacy owners may feel more enfranchised in the public health movement than their employees and locums. Indeed, one-in-ten locums reported that their job was not at all concerned with public health which, as locum pharmacists constitute over a third of actively employed community pharmacists, could be limiting factor in any drive to strengthen the public health function of community pharmacists

Mission description and in-flight operations of ERBE instruments on ERBS, NOAA 9, and NOAA 10 spacecraft

Instruments of the Earth Radiation Budget Experiment (ERBE) are operating on three different Earth-orbiting spacecraft. The Earth Radiation Budget Satellite (ERBS) is operated by NASA, and NOAA 9 and NOAA 10 weather satellites are operated by the National Oceanic and Atmospheric Administration (NOAA). This paper is the second in a series that describes the ERBE mission, and data processing and validation procedures. This paper describes the spacecraft and instrument operations for the second full year of in-orbit operations, which extend from February 1986 through January 1987. Validation and archival of radiation measurements made by ERBE instruments during this second year of operation were completed in July 1991. This period includes the only time, November 1986 through January 1987, during which all ERBE instruments aboard the ERBE, NOAA 9, and NOAA 10 spacecraft were simultaneously operational. This paper covers normal and special operations of the spacecraft and instruments, operational anomalies, and the responses of the instruments to in-orbit and seasonal variations in the solar environment

Developing the public health function of locum pharmacists under the auspices of the new pharmacy contract

Focal Point - There are reduced opportunities for locum pharmacists to access training and education that meets their needs and enables them to play a full role under the new pharmacy contract - Eighty-six per cent of locums consider themselves to be more health professional than business person, compared to just 48% of pharmacy owners - Forty per cent of locums believe that a lack of access to training is a major barrier to the development of their public health function - While locum pharmacists are arguably more likely to embrace 'professionalising', patient-care-based roles, they are also the group least likely to be able to access the necessary training to fulfill such roles Introduction It has been suggested that locum pharmacists do not want the business-based responsibilities (e.g. staff management, meeting targets, etc) that come with pharmacy management.1 Research also suggests that locums derive great satisfaction from the health-professional aspects of the pharmacists’ role (e.g. patient contact, the provision of advice, etc).1 However, upon the introduction of the new pharmacy contract (April 2005), concerns were expressed that it was becoming increasingly difficult for locum pharmacists to access training and education that would meet their needs and enable them to play a full role under the new framework.2,3 Method After piloting, in August 2006 a self-completion postal questionnaire was sent to a random sample of practising community pharmacists, stratified for country and sex, within Great Britain (n = 1998), with a follow-up to non-responders 4 weeks later. Data were analysed using SPSS (v12.0). A final response rate of 51% (n = 1023/1998) was achieved. Respondents were asked ‘indicate how you view yourself as a pharmacist’ – in terms of their relative focus on the health-professional and business aspects of their role. Respondents were also asked ‘do you consider a lack of training opportunities to be a barrier to the development of the public health role of community pharmacists?’. Results Locums were significantly more likely than owners or employees to consider each factor a major barrier. Discussion Four in 10 locums consider a lack of training opportunities to constitute a major barrier to the development of their public health function. Pharmacy may not be able to provide the services required of it by the policy agenda if pharmacists are unable to be involved in extended role activities through a lack of training opportunities. Therefore, the paradox that needs to be addressed is that while locum pharmacists are arguably more likely to embrace ‘professionalising’, patient-care-based roles, they are also the group least likely to be able to access training to fulfil such roles. The training needs of this large subset of the pharmacist population need to be assessed and met if the whole community pharmacy workforce is going to maximise its contribution to public health under the new contractual framework. References 1 Shann P, Hassell K. An exploration of the diversity and complexity of the pharmacy locum workforce. London: Royal Pharmaceutical Society of Great Britain; 2004. 2 Almond M. Locums – key players in workforce – cast adrift as contract launched. Pharm J 2005;274:420. 3 Bishop DH. A lack of appreciation of what really happens. Pharm J 2005;274:451

Community pharmacy in a commissioning-led NHS:can pharmacy compete effectively?

Introduction – The commissioning of services has been a core responsibility of English Primary Care Trusts (PCTs) since 2002. Primary care organisations (PCOs) in Scotland, Wales and Northern Ireland have also increased their commissioning activities but with, arguably, less fervour than their English counterparts. The commissioning function of English PCTs has been reinforced by the introduction of new contractual frameworks across primary care – for medical services, dentistry and pharmacy. The new pharmaceutical services contract for England and Wales introduced an “enhanced” category of services, the provision of which is dependent on the commissioning decisions of local PCTs. As the NHS, most pertinently in England, continues its transformation from a provider to a commissioner of healthcare, the ability of pharmacy to compete effectively for funding is likely to become increasingly important. Method - After piloting, in August 2006 a self-completion postal questionnaire was sent to a random sample of practising community pharmacists, stratified for country and sex, within Great Britain (n=1998), with a follow-up to non-responders 4 weeks later. Data were analysed using SPSS (v12.0). A final response rate of 51% (n=1023/1998) was achieved. Within the section of the questionnaire relating to service provision, respondents were asked “do you believe that pharmacy will be able to compete effectively with other healthcare providers for access to additional funding to develop services that address a public health need identified by your local Primary Care Organisation (PCO), e.g. PCT/LHB etc.?”. Answers were recorded on a three-point scale; pharmacy “will”, “may”, or “will not” be able to compete effectively for funding. Results - The attitudes of pharmacists showed variation depending on the type of pharmacy they worked in (supermarket, multiple (outlets (n)=200), large chain (200>n>20), small chain (20=n>5), or independent (n=5)) (?2 test with p=0.001). Over a third of survey pharmacists working in small chains and independents (37% (n=21/57) and 33% (n=113/341) respectively) believed that pharmacy would not be able to compete effectively for funding compared to 23% (n=15/65) for supermarket pharmacists, 22% (n=21/97) for pharmacists employed by large chains and just 18% (n=62/353) for pharmacists employed most regularly in multiples. Furthermore, attitudes also varied between the countries of residence of respondents (?2 test with p<0.05). 27% (n=242/893) of pharmacists resident in England and Wales believed that pharmacy would not be able to compete compared to 16% (n=18/116) of pharmacists resident in Scotland. Conclusions – It would appear that community pharmacists believe that the larger pharmacy chains and supermarkets will occupy an advantageous position in terms of attracting finance to develop services. This could have notable implications for service provision across the sector. If corporate pharmacy chains were to monopolise commissioning monies then the proportion of funding available to independents will be diminished; arguably further hastening their demise, as well as stifling the professional development of pharmacists employed within the independent sector. These findings, when combined with the variation observed between UK pharmacists operating under different contractual frameworks, may be a reflection of the divergent policy in the different administrations with developments in England, including the new pharmacy contract, reflecting a market-based approach with Scotland taking a near opposite stance with service integration and a commitment to new public health. However, it should be acknowledged that the questionnaire did not allow for detection of ambiguities in, or misunderstandings of, the survey question and this should be considered as a limitation of the research

Brain States That Encode Perceived Emotion Are Reproducible but Their Classification Accuracy Is Stimulus-Dependent

The brain state hypothesis of image-induced affect processing, which posits that a one-to-one mapping exists between each image stimulus and its induced functional magnetic resonance imaging (fMRI)-derived neural activation pattern (i.e., brain state), has recently received support from several multivariate pattern analysis (MVPA) studies. Critically, however, classification accuracy differences across these studies, which largely share experimental designs and analyses, suggest that there exist one or more unaccounted sources of variance within MVPA studies of affect processing. To explore this possibility, we directly demonstrated strong inter-study correlations between image-induced affective brain states acquired 4 years apart on the same MRI scanner using near-identical methodology with studies differing only by the specific image stimuli and subjects. We subsequently developed a plausible explanation for inter-study differences in affective valence and arousal classification accuracies based on the spatial distribution of the perceived affective properties of the stimuli. Controlling for this distribution improved valence classification accuracy from 56% to 85% and arousal classification accuracy from 61% to 78%, which mirrored the full range of classification accuracy across studies within the existing literature. Finally, we validated the predictive fidelity of our image-related brain states according to an independent measurement, autonomic arousal, captured via skin conductance response (SCR). Brain states significantly but weakly (r = 0.08) predicted the SCRs that accompanied individual image stimulations. More importantly, the effect size of brain state predictions of SCR increased more than threefold (r = 0.25) when the stimulus set was restricted to those images having group-level significantly classifiable arousal properties

Impact of asthma on educational attainment in a socioeconomically deprived population: a study linking health, education and social care datasets

BACKGROUND: Asthma has the potential to adversely affect children's school examination performance, and hence longer term life chances. Asthma morbidity is especially high amongst UK ethnic minority children and those experiencing social adversity, populations which also have poor educational outcomes. We tested the hypothesis that asthma adversely affects performance in national school examinations in a large cohort from an area of ethnic diversity and social deprivation. METHODS AND FINDINGS: With a novel method (using patient and address-matching algorithms) we linked administrative and clinical data for 2002-2005 for children in east London aged 5-14 years to contemporaneous education and social care datasets. We modelled children's performance in school examinations in relation to socio-demographic and clinical variables. The dataset captured examination performance for 12,136 children who sat at least one national examination at Key Stages 1-3. For illustration, estimates are presented as percentage changes in Key Stage 2 results. Having asthma was associated with a 1.1% increase in examination scores (95%CI 0.4 to 1.7)%,p = 0.02. Worse scores were associated with Bangladeshi ethnicity -1.3%(-2.5 to -0.1)%,p = 0.03; special educational need -14.6%(-15.7 to -13.5)%,p = 0.02; mental health problems -2.5%(-4.1 to -0.9)%,p = 0.003, and social adversity: living in a smoking household -1.2(-1.7 to -0.6)%,p<0.001; living in social housing -0.8%(-1.3 to -0.2)% p = 0.01, and entitlement to free school meals -0.8%(-1.5 to -0.1)%,p<0.001. CONCLUSIONS: Social adversity and ethnicity, but not asthma, are associated with poorer performance in national school examinations. Policies to improve educational attainment in socially deprived areas should focus on these factors